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Description 

This invention relates to audio/visual compilation in 
data processing systems enabling computer-based pro- 
duction of audio/visual presentations, and seeks to im- 
prove the software-supported user interface for enabling 
a personal computer to produce and display an integrat- 
ed audio/visual presentation. 

As personal computers (PCs) have improved in 
processing capability, various application programs 
have been developed which enable the creation and dis- 
play of audio/visual presentations. In the mid 1980's, the 
IBM Corporation marketed a PC application program en- 
titled: "PC Storyboard" which enabled the preparation 
and display of colour presentations on an IBM PC. The 
PC Storyboard software was comprised of four program 
segments. A first segment, entitled: " Picture Maker", en- 
abled the creation and modification of pictures in medium 
resolution graphics. Picture Maker included commands 
to write text, drawfigures, generate charts and to cut and 
paste images between pictures. A second segment, 
"Picture Taker", was employed to capture picture images 
of scenes from other PC application programs. "Story 
Editor' , was a third segment which enabled the PC user 
to organise pictures into presentations (stories). It pro- 
vided for the selection of a variety of picture-to-picture 
transition methods that allowed one picture to dissolve 
into another. Variables such as display times, colours 
and whether the picture would be developed as a full pic- 
ture or as a series of partial pictures, were also enabled 
by this software. Storyboard also included a segment en- 
titled: "Story Tell", which enabled the presentation of sto- 
ries assembled by the other software segments. 

While Storyboard was, for its time, an acceptable 
product for the PC, it lacked a number of capabilities. It 
was essentially a visual presentation assembler and pre- 
senter; it lacked any significant capability for audio pres- 
entation assembly; for synchronisation of an audio pres- 
entation with the visual presentation; for interspersing 
during the presentation, commands which enabled logi- 
cal and data processing interactions between the viewer 
and the PC; etc. 

Recently, the increased use of windows, pull-downs, 
advanced cursor-selection techniques and other dis- 
play-oriented, user interface instrumentalities have 
come into favour. These enable a PC user to directly in- 
terface with a PC's software and to control it largely from 
cursor-controlled screen selections. Substantial capabil- 
ity, colour presentation systems with synchronised audio 
have not, to the Inventor's knowledge been made avail- 
able for the PC market. 

In addition to PC Storyboard, other prior art has pro- 
vided audio/visual, computer-based presentation capa- 
bilities. For instance, in U.S. Patent 4,71 2,180 to Fujiya- 
ma et al, a computer-assisted instruction system is de- 
scribed wherein audio and pedagogy information are 
compiled to produce a recording medium with an inte- 
grated mode of presentation, various embodiments are 



described for integrating the audio and non-audio com- 
ponents of the presentation. In U.S. Patent 4,645,459 to 
Graf et al, the use is described of a library of images to 
produce improved quality "real-world" images for com- 

5 puter imagery. Lemelson, in U.S. Patent 4,646,172 de- 
scribes a video presentation system wherein audio and 
video segments are combined to produce an integrated 
presentation. None of the above references teach a us- 
er-friendly interface which enables direct and simple 

10 methods for compilation of the presentations. 

In U.S. Patent 4,746,994 to Ettlinger, an editing sys- 
tem is described wherein selected film segments may be 
spliced independent of picture and sound track sources 
and timing. The system provides a graphically arranged 

75 representation of the script of the work being edited, thus 
permitting the editor to preview and select segments of 
different "takes". This system attacks considerably dif- 
ferent problems than those encountered in compu- 
ter-based audio-visual presentations. 

20 EP-A-239884 discloses an interactive video compo- 
sition and presentation system which allows for the spec- 
ification and execution of independent, multi-media 
tasks along a synchronising time-line, preferably using 
a spread-sheet matrix. 

25 TTie present invention provides a method of creating 
and performing a synchronised, audio/visual presenta- 
tion in a data processing system having audio and visual 
input/output, the method comprising storing a plurality of 
visual images; and characterised by: displaying a table 

30 representing an audio sound-track together with corre- 
sponding time indications, and entering synchronisation 
labels at selected time indications; entering a command 
for displaying selected stored visual images and associ- 
ating said command with a synchronisation label in the 

35 audio sound-track table; and executing the audio/visual 
presentation by playing the audio sound-track, and re- 
sponsive to recognising a synchronisation label therein, 
invoking the command associated with said synchroni- 
sation label to display the selected stored visual image. 

40 Such an arrangement is thought to provide a com- 
puter-based audio/visual assembly and presentation 
system which efficiently exhibits visual presentations 
with synchronised audio. 

There is disclosed hereinafter, by way of example, 

45 a user/PC interface which enables the creation and per- 
formance of a synchronised audio/visual story on the PC. 
The interface initially enables the storage of a plurality of 
visual images. Then, it enables the creation of an audio 
presentation which includes labels and time indications, 

so certain of which are employed for synchronisation pur- 
poses. Means are provided in the computer which are 
responsive to a label to execute a predetermined com- 
mand upon the appearance of a label in the audio pres- 
entation. The computer/software system is then operat- 

55 ed to perform the audio presentation, that performance 
automatically causing the display of visual images upon 
the occurrence of labels and time indications. 

It incorporates an improved, table-based authoring 
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system is also described for preparation of both the 
above noted audio and visual presentations. The system 
relies upon columnar presentations of a tabular form, 
each column indicating a separate control function, com- 
ment, or other free form statement relating to the au- 
dio/visual presentation. This combination enables inter- 
active processing with a user during the presentation and 
which is adapted to run on a PC and provides an au- 
dio/visual synchronised presentation system for a PC 
wherein intelligent interfaces on the PC screen enable a 
user to easily employ the system. 

The present invention will be described further by 
way of example with reference to an embodiment thereof 
as illustrated in the accompanying drawings, in which 

Fig. 1 is an illustration of the software segments 
used; 

Fig. 2 shows a library table screen used; 

Fig. 3 shows a library application table screen used; 

Fig. 4 shows menus which are displayed when either 
Prompt or Process actions on a screen "action bar" 
are selected on a library application screen; 

Fig. 5 shows menus which are displayed when either 
Edit or Inquire actions on a screen "action bar" are 
selected on a library application screen; 

Fig. 6 shows the menus which are displayed when 
either Print, Exit or Help actions on a screen "action 
bar" are selected on a library application screen; 

Fig. 7 illustrates a screen presentation of a first level 
library table showing various applications available 
to the audio/visual system; 

Fig. 8 is a display of a "bill of materials 0 which 
appears on the screen when one of the applications 
shown in Fig. 7 is selected; 

Fig. 9 is another "bill of materials" displayed on the 
screen and indicates the use of reference address- 
ing; 

Fig. 10 is a screen exhibiting the audio editor table; 

Fig. 11 is a showing of the menus which appear 
when either the Prompt, Audio or Edit indications in 
the "action bar" of Fig. 10 are selected; 

Fig. 12 is a showing of the menus which appear 
when either the Mix, Controls or Print indications in 
the "action bar" of Fig. 10 are selected; 

Fig. 13 is a showing of the menus which appear 
when either the File, Exit or Help indications in the 



"action bar" of Fig. 10 are selected; 

Fig. 14 is a showing of an audio editor table which 
has been compiled to present an audio piece; 

5 

Fig. 15 and 16 are high level flow diagrams helpful 
in understanding the user/system interactions 
required to produce the table of Fig. 14; 

10 Fig. 17 is a screen showing of a story editor table 
used; 

Fig. 1 8 is a showing of the screen of Fig. 1 7 with the 
five right-most columns of the story editor table 
is replaced with additional category columns; 

Figs. 19-27 indicate the screen type line menus 
which appear for each column of the story editor 
table of Fig. 14; 

20 

Figs. 28-35 indicate sound type line menus which 
appear for each column of the story editor table of 
Fig. 14; 

25 Fig. 36 is a showing of the menus which appear 
when either the Prompt or Tell actions are selected 
on the story editor table "action bar"; 

Fig. 37 is a showing of the menus which appear 
30 when either the Edit or View actions are selected on 
the story editor table "action bar"; 

Fig. 38 is a showing of the menus which appear 
when either the Print or File actions are selected on 
35 the story editor table "action bar"; 

Fig. 39 is a showing of the menus which appear 
when either the Exit or Help actions are selected on 
the story editor table "action bar"; 

40 

Fig. 40 shows prompt tables which appear when the 
Prompt indication is selected on the story editor 
"action bar"; 

45 Fig. 41 shows AVA (audio/visual authoring lan- 
guage) functions which appear when the AVA func- 
tions line in Fig. 40 is selected; 

Fig. 42 shows the AVA commands which appear 
so when the AVA commands line in Fig. 40 is selected; 
and 

Fig. 43 is a showing of a story editor print-out illus- 
trating an audio/visual presentation. 

55 

It is to be understood initially that the invention may 
be configured from either software or firmware, either of 
which may be adapted to run on a PC, such as the IBM 
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PS/2. As is known, most modem PCs are constructed to 
employ a bus structure with PC/subassembly communi- 
cations taking place, in the main, over the bus or buses. 
Characteristically, a graphics capable PC is provided 
with an arithmetic/logic unit, random access memory, 
various disk drives for storing quantities of readily avail- 
able data, and a colour display including a keyboard. The 
display is adapted to show an image or images, along 
with a cursor (or cursors) which enable user selection ot 
various software subroutines. All of the aforementioned 
structure is conventional and knowledge ot its operation 
is assumed in the succeeding description. 

Referring now to Fig. 1 , the six major software seg- 
ments of the invention are schematically shown. The 
overall program includes a library editor; image editor; 
digitise editor, convert editor audio editor; and a story 
editor. During the course of the description, the digitise 
and convert editors will only be briefly relerred to, where- 
as the others will be discussed in detail. 

In brief, the library editor manages the retrieval, 
processing and storing of audio-visual "objects" (i.e., an 
■object" is an image, audio file or story used in the crea- 
tion of an audio/visual presentation). The image editor is 
primarily used to add text and draw graphics onto imag- 
es; to define special field types such as input or cursor 
actuation fields, and to do other image manipulations 
such as resizing or rotating. The digitise editor is used 
for converting video camera (analog) input images into 
computer digital form. These images are then stored as 
files in the PC where they become available for use in 
the audio/visual operation. The convert editor is used to 
modify images captured by scanners or other image dig- 
itising systems and to convert them into a form usable in 
the audio/visual system of this invention. The audio ed- 
itor enables the recording (digitising) and editing of 
sound. It also enables the insertion of time and other sig- 
nals which are subsequently used to control the presen- 
tation of the story. The story editor assembles audio/vis- 
ual presentations (called stories) and pulls together all 
of the data from the keyboard, files and various editors 
to present a completely synchronised, audio/visual pres- 
entation. 

LIBRARY EDITOR 

Each time the software system of this invention is 
initially operated, library editor is automatically invoked 
as the first editor accessed. The library editor provides 
access information on all objects within the software seg- 
ments and their paths. The user selects a particular au- 
dio/visual object to work with (such as an image or a sto- 
ry), and then selects the type of processing to perform 
on the object (such as edit, digitise, convert, etc.)- These 
choices automatically determine which of the other edi- 
tors is to be invoked. Thus, the other editors are invoked 
according to the object and task selected rather than by 
an explicit editor name. The selected object is automat- 
ically retrieved for the invoked editor at the same time 



the editor is activated, so that the object is present for 
use as soon as the editor comes on-line. 

At the conclusion of processing by a respective ed- 
itor, control returns to the library editor, with any object 

s changes now saved to disk. Once back in the library ed- 
itor, the user can select either another object or object 
processing or can exit to other operations. Thus, move- 
ment to and from editors is normally routed through the 
library editor. In essence, the library editor is the "main 

10 menu" from where branching to all individual object ed- 
itors occurs. 

All objects, such as images, stories and audio files 
are listed in the library editor's displays. It is to be remem- 
bered that it is these object lists that are used by the sys- 
is tern to cause various object editors to be invoked and it 
is the paths to the objects indicated in these lists that are 
used to control where objects are retrieved from and 
saved to. 

While the library within the library editor contains in- 

20 formation about various directories and objects, (e.g., 
type, name, description and path), it does not contain the 
directories or objects themselves. It essentially is an in- 
dex which points to an object, wherever that object may 
reside within the system. 

25 The term "application" will be used hereinbelow and 
it is meant to refer to a collection of objects grouped to- 
gether in the library in a way that is meaningful to the 
user. An application may be a series of objects usable 
for a product presentation, an advertising message, or 

30 another audio/visual presentation. The application will 
have a name and be listed and accessed by that name. 
Furthermore, each application will have a secondary list 
embedded in it which, in addition to indicating the partic- 
ular application, lists all of the various objects which are 

35 part of the application (i.e., a "bill of materials" for the 
application). 

In the discussion below, a plurality of display screens 
will be described which enable the principal modes of 
interaction between the user and the system. Each 

40 screen presents a table, window, pull-down, action bar 
or combination thereof which provides for information 
feedback from the system to the user and vice-versa. 
Screens are generally provided with a plurality of cursor 
kinds (e.g., highlight bars, underlines, emphasis marks, 

45 etc.) to enable a user to know which portion of a screen 
is enabled for an input. Some cursor kinds automatically 
move on the screen (e.g., while a software segment is 
"playing",) or are moved by the user's actuation of a 
mouse, keyboard keys (e.g., right, left, up, down arrows, 

so tab key, backspace, etc.) or other cursor control instru- 
mentalities. 

Turning now to Fig. 2, a first level library editor 
screen entitled "library table" is illustrated. This table lists 
the various applications and their names that are con- 
55 tained within the library editor. It forms the highest level 
directory to all subsidiary directories related to the vari- 
ous listed applications. An "action bar" 10 appears on 
line 1 of the screen and contains a number of action in- 
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dications. The placement of a cursor at any listed action 
indication will invoke a pull-down with a listing of the 
available functions pertaining to the chosen action. 

On line 3 of the screen, the term "library" indicates 
that this screen may be accessed by typing "library" fol- 
lowed by its particular name. Since the library editor can 
contain hundreds of listed applications, the combination 
of "library" with an application's name provides the sys- 
tem with the input necessary to identify the correct library 
screen. Assuming that the PC operates with a disk op- 
erating system (DOS), each library is physically one 
DOS directory. 

The library table of Fig. 2 comprises a plurality of 
columns. The left-most column (FR) will have either an 
F or an R adjacent a line. F indicates the application is 
physically resident in the currently connected file. R in- 
dicates that the named file is located in another file not 
connected at the moment. The next column to the right 
"Type", defines the kind of file indicated on the line. In 
most cases the indication will be "application". The next 
column (headed by D:) indicates, if a D is present in a 
line, that the file is on a drive not presently on-line. The 
Name column indicates the assigned name of the appli- 
cation and the Comments column includes any descrip- 
tive comments regarding the file as well as a path indi- 
cation if an R is present in the left-most column. 

In Fig. 7, an example of a library table is shown with 
certain of its columns filled in. Note that all of the listed 
files are application files; that the last two application files 
are housed on drive e (which is not on-line); that the 
names of the files are as indicated; and that the adjacent 
comments describe, briefly, the subject matter of each 
application. For instance, Demo 1 presents the "current 
product line" as an audio/visual presentation. 

Each line within a library table indicates the name 
and gross contents of the application. In order to deter- 
mine the detailed contents of an application, the "appli- 
cation" table associated with the particular application 
must be accessed. This action will bring up for view, a 
"bill of materials" which lists all of the objects associated 
with the application. 

An application table may be selected by placing the 
cursor on the line of the library table where it appears 
and selecting, from action bar 10, a processing action to 
perform (e.g., edit). When an application is selected, the 
system retrieves an existing application from the file (or 
enables the creation of a new one, if that is desired) and 
then displays its corresponding application table screen 
which lists all images, stories, audio files, etc. associated 
with the application. 

Turning to Fig. 3, a representative application table 
is shown. This table, as aforestated, displays the con- 
tents of one specific application. Like the library table, 
the application table contains on its first line, an action 
bar 10 containing a number of indicated actions which, 
when invoked, cause the display of detailed pull-downs. 

Line three contains the indication "application" along 
with the name of the application. The left most table col- 



umn (FR) indicates if the object listed on the line is phys- 
ically resident in this application. An F indicates the ob- 
ject is resident in the application file. If an R (for "refer- 
ence") is shown, it indicates that the object listed is not 
5 physically resident in this application file. A reference to 
where that object is actually stored is found in the 
right-most columns of the table (i.e., D: Reference). 
There is inserted path information pointing to the location 
where the object is stored. A blank in the FR column sig- 
nifies the line is a comment-only type line. 

The Type column denotes whether the object is an 
image, audio, story or other object. The Name column 
contains a user-assigned name for the object. The col- 
umn to the right of the Name column includes comments 
which describe something about the object to enable the 
user to recall its substance. For example, as shown in 
Fig. 8, an exemplary application table is illustrated hav- 
ing three image objects, two audio objects and one story 
object. Each image object has an assigned name (e.g. 
"Model 1 ") which enables that object file to be identified. 
A comment next to each object listing explains some- 
thing about the object. 

It is from an application table screen that access to 
the object editors occurs. While not shown, a cursor is 
present on the screen and is free to roam, under user 
control, from line to line and scroll from page to page. To 
process an existing object, the cursor is placed on the 
line for the object and an action from the action bar is 
invoked. This results in display of a pull down listing of 
available functions with the action classification. If then, 
a cursor is placed next to a function (e.g., "edit file") and 
a selection made, the proper editor program segment is 
accessed. For instance, the "edit file" action invokes the 
audio editor for an audio object, image editor for an im- 
age object, and story editor for a story object. 

Turning to Figs. 4-6, the pull-downs (and related 
functions) which result from the invoking of one or more 
actions in action bar 10 are illustrated. As shown in Fig. 
4, a cursor indication placed at the Prompt location in 
action bar 10 and a user actuation of a Prompt -associ- 
ated keystroke (e.g. by typing P) causes pull-down 12 to 
appear. This pull-down both shows and enables the 
available entries which can be made in the "type" column 
at a selected line in an application table. 

Placing the cursor at the Process location in the ac- 
tion bar enables pull-down 14 to be displayed. If the cur- 
sor is then placed next to the "edit file" line and the ALT 
and E keys depressed by the user, an appropriate editor 
is selected as above described. This, then, enables the 
particular object listed in the application table to be ap- 
propriately altered by one of the editor program seg- 
ments. The "run program" line is particularly related to a 
listed story object. The remaining listings shown in 
pull-down 14 are self-explanatory. 

In Fig. 5, the Edit action pull-down is shown as well 
as the Inquire action pull-down. In Fig. 6, the Print, Exit 
and Help pull-downs are shown. Each of the lines within 
each of the pull-downs defines a particular task which is 
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enabled upon the appropriate positioning of a cursor and 
selection of the task. For instance, in the Edit pull-down, 
placing a cursor next to the "undo" line and user depres- 
sion of an Enter key removes the last edit change that 
affected the application table screen, such as an insert 
or a move. 

Referring now to Fig. 9, an application table is shown 
having a number of referenced objects listed therein. As 
aforestated, references allow objects to be associated 
with an application without physically residing in the 
same directory. For example, assume a user keeps ail 
audio music stored in a single application/directory such 
as, MYAUDIO. This is attractive because all music is in 
one place for convenient access, maintenance and 
backup. If now, for various demonstration applications, 
different pieces of music are desired, there are a number 
of approaches possible to link the demonstration and the 
music together. The music could be physically copied 
into the different directories. This would be unacceptable 
as much duplicate storage and multiple copies would be 
required. Alternatively, the different demonstration sto- 
ries could place specific path information with every 
"play" type statement. This would avoid storage duplica- 
tion since the demonstration applications would still be 
using the original audio versions. However, if any soft- 
ware path change needed to be made to the MYAUDIO 
file (e.g., if it was placed on a different drive), all path 
entries on "play" statements related to the music would 
require finding and changing. 

References in this invention work around these 
drawbacks by allowing an object to be listed in an appli- 
cation as if it were physically there. Path information is 
provided in the right-most column of the application table 
and redirects access to the application and file where the 
object is actually physically located. In Fig. 9, examples 
of referenced paths are shown. The referenced object, 
SKIING is listed as an image object. This enables any 
"show* statement in a story working from this application 
to use SKIING as an object name without requiring any 
special path designation - exactly as is done for any ob- 
ject physically stored in this application such as the file 
INTR0 1 . At story run-time, when a play statement is en- 
countered with SKIING, the program automatically caus- 
es the path "e: IMAGES 4" to be searched for the image 
object SKIING. Hence, while SKIING both can be played 
and edited when working from this application, the actual 
physical file is continually retrieved from and stored in 
the referenced application/directory e:IMAGES 4. Thus, 
only one path designation need be maintained for SKI- 
ING - the one on the object line in the application table 
-- where SKIING is actually resident. 

When applications being referenced all reside 
on-line, the benefit from the reference procedure is that 
duplicate storage is not needed -- only one copy of an 
object need be maintained and other applications out- 
side of the one owning the object can access it by point- 
ing to it (referencing it). Another benefit is that these path 
references do not have to be repetitively entered and 



spread throughout for each object. Each application 
needs just one reference (path designation) per each ob- 
ject referenced. Thus, this inables one object copy to be 
accessed by all. 
5 Once a user has fully assembled all of the objects 
required to produce an audio/visual presentation, the 
next action required is to assemble the audio portion of 
the presentation. This is accomplished in the audio edi- 
tor. 

10 

AUDIO EDITOR 

The audio editor is used to record, digitise, and edit 
sound. All work done with the audio editor is done 

is through a columnar table displayed on the PC screen 
and shown in Fig. 10. Sound is handled in time lines of 
one second increments. A volume level is indicated for 
each sound line. Each line may also contain comments 
for annotation, sound controls for volume adjustment 

20 and labels for story synchronisation. It is also possible to 
have comment lines in the table without any accompa- 
nying sound. Such comment lines may be used to doc- 
ument full audio scripts. Edit action such as copy, lift, 
paste, insert and delete allow audio to be cut, reordered 

25 and spliced via software controls. A zoom to tenths of a 
second facilitates extremely fine editing, if desired. 

As with the previously table screens, the audio editor 
table of Fig. 10 includes an action bar 20 which is used 
to invoke various pull-downs and associated subrou- 

30 tines. The name of the current audio object in process is 
displayed following the audio A label. Audio B provides 
for a second channel of audio, and if in use displays the 
audio object B name. During the operation of the audio 
editor table, a background colour cursor 22 is employed 

35 and is enabled to scroll up or down in accordance with 
user actuated keyboard controls. During playing and re- 
cording, cursor 22 extends across the full width of the 
table and keeps pace, on a line for line basis, as music 
is recorded or played. At other times during the opera- 

40 tion, only a portion of cursor 22 is used to indicate the 
particular column enabled for input from the user. 

In the audio editor table, the left-most column "Play- 
time" contains a series of numbers from 0000 to 9999 to 
represent which second of sound is indicated. Seconds 

45 are numbered consecutively for all sound lines. Thus, 
any insertion or deletion of sound causes an appropriate 
renumbering of the column. In essence, the column 
gives an indication, from the beginning of Playtime to the 
end of Playtime, of the total number of seconds expend- 

50 ed and, more particularly, an indication of each individual 
second. A line having no sound associated with it is a 
"comment only" line and 3 dashes are displayed in the 
Playtime field. 

The "Sound" column field contains a plurality of 

55 squares sequenced left to right indicating the volume lev- 
el of this sound second. Each square represents an ad- 
ditional level of volume. By referring to Fig. 1 4, examples 
of these squares in the sound column can be seen. The 
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next column ("Synch Label") accepts a label which is 
adapted to be referenced in the story editor table (to be 
described below) as an audio point to synchronise on. In 
essence, a synch label may be a fundamentally fixed au- 
dio point to synchronise to, as contrasted to time values 
which are more relative in that they may change when- 
ever an audio object has sound inserted or deleted. It is 
to be noted that not alt synch labels are for system syn- 
chronisation, but may be used simply as prompts to en- / 
able the user to insert other sound or material in the audio 
track. 

The comments column is used to identify musical 
pieces, sections, highlights, and to enter voice portions 
or written scripts. Thus, as cursor 22 scrolls downwardly 
in the audio editor table in time with line-by-line music 
play, the user is enabled to know when to commence 
insertion of voice-over portions. In addition, the exact 
script is highlighted by cursor 22 as it scrolls downwardly. 

The right-most columns in the audio editor table en- 
able the control of the sound's volume and duration time. 
The method column only accepts the entry ■volume". A 
user may key this in, or select it from the prompt list (see 
Fig. 1 1 ). During play of audio, this column enables a vol- 
ume change according to the parameters noted in the 
two columns to the right (volume and time). As shown in 
Fig. 11 , (leftmost portion), the sound volume can be var- 
ied from zero percent to 100 percent and the time that it 
takes to move between volumes is indicated by the 
amount of time in the "time" column. Thus, a volume 
change can occur immediately (zero seconds), or can 
take up to ten seconds to change from one volume to 
another volume. The volume adjustments do not perma- 
nently affect the digitised sound unless the user elects 
to have them do so, in which case the user invokes the 
"mix" action from action bar 20 and further selects the 
"permanent sound mix" listing thereunder. 

The screen display of the audio editor table may be 
divided into top and bottom sections, when set for a dual 
view of audio objects A and B (not shown). The top sec- 
tion scrolls audio object A, and the bottom section scrolls 
audio object B. The active cursor may be moved back 
and forth between the two sections via a cursor toggle 
action. When the active cursor is highlighted in one sec- 
tion of the split screen, the other section (the other object) 
has an "echo cursor" which tracks along in synchronisa- 
tion. When the object with the active cursor is scrolled, 
by simply moving the cursor or performing an action such 
as, play, record, insert, etc., the other screen's cursor is 
normally scrolled in synchronisation. This, thus enables 
the two audio objects to be mixed by the user viewing 
these screens for synchronisation and control purposes. 

Turning now to Figs. 11,12 and 13, the pull-downs 
associated with the actions in action bar 20 will be briefly 
considered. The Prompt action has been described 
above under the Audio action. A pull-down displays a 
plurality of play commands which enable the playback of 
either an entire audio piece or a portion of a piece, the 
repeat of a piece or a specific time segment of a piece. 



For instance, the Play command causes the playing of 
an audio object starting at the current cursor location in 
the audio table. As play progresses, the highlight cursor 
keeps pace through the audio table, line by line. Lines 

5 with no sound associated with them (comment only lines) 
are skipped over. Play continues until either an escape 
key is depressed or the last line of audio is reached. The 
re-record entry in the Audio action pull down enables ad- 
ditional recordings to be made of audio objects. 

10 Edit actions, as indicated in the associated 
pull-down, enable a reversal of the last entry (undo) that 
affected the audio. The copy, lift and paste functions re- 
quire the use of a buffer to temporarily store the material 
being altered while it is being operated upon, in accord- 
's ance with the desired stated function. For instance, the 
paste command takes material from one portion of the 
table and puts it in another portion of the table. In Lift, 
the block to be "lifted" is placed in the buffer and is de- 
leted from the audio table. 

20 The remaining functions shown in Figs. 12 and 13 
are employed to enable alteration of the entries in the 
audio editor table for the purpose of creating a continu- 
ous audio object which can either be played alone or 
combined with a story as an integrated presentation. 

25 As shown in Fig. 1 2, the Mix, Controls and Print ac- 
tions in action bar 20 enable various mixing functions to 
be carried out between A and B sound sources; enable 
various controls to be "manipulated" with respect to 
sound volume and balance; and enable the printing of 

30 all or a portion of the audio editor table. In Fig. 13, the 
File, Exit and Help actions respectively enable the writing 
to disk of one or more audio selections; the exiting from 
the program upon certain conditions and the accessing 
of help subroutines which enable the user to overcome 

35 encountered problems. 

AUDIO AUTHORING 

Referring now to Figs 1 4, 1 5, and 1 6, the preparation 

40 of an audio editor table for an actual presentation will be 
described. As shown in the flowchart of Figs. 15 and 16, 
the user initially enters the library editor; selects an ap- 
plication, and then selects an audio object. The Process 
action in the action bar is then selected and the associ- 

*s ated pull-down appears with the "edit file" function indi- 
cated. That function is selected and causes the audio 
edit table to be displayed. At this stage, the user selects 
the sound source (either A or B channel), places the cur- 
sor at the audio object name in the table and invokes the 

so Audio action bar and its associated pull down appears 
(see Fig. 11). By then placing the cursor at "play from 
beginning and depressing the Alt and G keys, playing of 
the audio object commences. 

At the beginning of the audio object play, the audio 

55 editor table is the blank table as shown in Fig. 1 0. As the 
audio object commences play, cursor 22 begins to scroll : 
on a second by second basis, down the table. The music 
then plays in second by second increments, the seconds 



13 



EP 0 403 118 B1 



14 



are sequentially marked and volume indications of the 
sound are inserted in the sound column. 

All of the above actions are noted in the audio table 
- see Fig. 1 4 where an example of an actual presentation 
is shown. Wherever there is no sound, three dashes are 
inserted in the play time column. (Some of the seconds 
indications have been removed from the chart of Fig. 1 4 
for brevity's sake.) The music chosen to play is "Anchors 
Away" and is enabled to play for 22 seconds as an intro- 
duction to commentary. At the 20 second time line, an 
insert is made to the sound control columns to indicate 
that the volume is to decrease to 20% over a two-second 
period. Play then continues at the 20% level until the ter- 
mination of the piece or an earlier time (e.g. second 42). 

At the end of the audio piece, the user moves the 
cursor to where the voice over text is to be inserted. The 
text may then be typed into the comment area or it can 
be directly recorded. Additionally, synch labels may be 
inserted in the synch label column for the purpose of user 
prompting or to provide a control signal for a story editor 
table (described below). In Fig. 14, at second 34 it is de- 
sired to cause a specific "show" to occur, synchronised 
with the inserted commentary. Thus, a synch label 
NORD is inserted which can then be referenced by the 
story editor table as a control function. The remaining 
synch labels are prompts for the user to know when cer- 
tain actions are to take place during the running of the 
audio table. For instance, the term fleet" is the cue to 
commence the recording of the first comment line shown 
in the comment column. The subsequent label "shore 
party", enables the user to know that a picture of a group 
of sailors will be shown at this point during the presen- 
tation. 

As shown in Figs. 15 and 16, once the synch labels 
are inserted, volume commands can also be inserted 
and the voice-over recorded in the audio B area. Subse- 
quently, the A and B audios can then be mixed to create 
a single audio record and the synch labels and volume 
commands modified, if necessary, to cause the correct 
balance of sound between the background music and 
the voice-over. 

STORY EDITOR 

Story editor is used to assemble audio/visual pres- 
entations (i.e. a story). A story defines the sequencing 
and display of images and the synchronisation of the au- 
dio. Data input from the keyboard and data outputs to 
the screen and to files are also controlled by the story 
editor. To build a story, story editor employs an authoring 
language called AVA (Audio/Visual Authoring language) 
and a specification table that is very much like that used 
with PC Storyboard. The AVA language adds an interac- 
tive and data processing capability to the invention that 
was not available in PC Storyboard. Overall, the AVA lan- 
guage has some 60 instructions available for use. A list- 
ing of the AVA statements, functions and commands is 
illustrated in Figs. 40, 41 and 42. It should be noted that 



the AVA language is largely based on REXX, (Restruc- 
tured, Extended Executor) a command language used 
in IBM products. That language is defined in "System 
Product Interpreter Reference", Second Edition, Dec. 
s 1 984, IBM publication number SC245239-1 . 

When displaying images in the story, either the en- 
tire image can be displayed all at once or just a portion. 
Timing specifications in the story control how long imag- 
es display, how long the transition takes from one image 
10 to another, and synchronisation between images and au- 
dio. Synchronisation originates with the audio editor, to 
which the story editor responds. Stories can be set to 
test, at run time, for the occurrence of an event e.g. a 
user's actuation of a cursor field. Such fields are defined 
is on images (such as around icons or key text words) and 
result in a special indication when selected by a cursor 
or mouse at story run -time. In this fashion, a screen of 
icons can be displayed for selection and upon user re- 
sponse, the story can determine which icon was selected 
20 and then act (i.e. branch) accordingly. 

Referring now to Fig. 17, a story editor table (no. 1 ) 
is illustrated. In Fig. 18 t story editor table No. 2 is illus- 
trated and it will be noted that only the right- most four 
columns differ from that of table 1 . In essence, the story 
25 editor table is comprised of 1 1 columns; however only 7 
columns can be shown in one display screen, while the 
remaining four columns need to be relegated a second 
screen (Fig. 18). 

Each column of the story editor table has assigned 
30 to it, a preset number of character positions. Ordinarily, 
entry of data in one column is restricted to the preset 
number of assigned characters. If an attempted entry is 
made of a greater number of characters, the "overage" 
characters are neither displayed nor accepted. If it is de- 
35 sired to insert a statement or comment which exceeds 
the number of character positions available in any col- 
umn, the insertion of a slash followed by an asterisk in 
the first two character positions of a tine overrides all sub- 
sequent column character limitations and enables text to 
40 be inserted for the full length of the line. 

To enable ease of movement between columns, tab 
codes are emplaced at the first character position of each 
column so that by depression of a tab key, the cursor can 
be incremented across a tine on a table from the first 
45 character position of a column to the first character po- 
sition of the next column etc. If however, the cursor is in 
the first character position of the right-most column (size) 
of the table shown in Fig. 1 7 and the tab key is pressed, 
the table shown in Fig. 18 is automatically displayed in 
so place of table of Fig. 1 7. It is to be noted that the left -most 
two columns of both tables are identical. 

The story editor tables shown in Figs 1 7 and 1 8 use 
a cursor which is represented by a bright flashing under- 
score that appears under whatever character position 
ss the cursor is currently located at. In addition, a colour 
band entirely fills the background of the field within a line 
being addressed. In many cases, a column and a field 
are synonymous. For example, if the cursor is in the time 
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column, only that field on the current line has a back- 
ground colour bar. When the current line is an AVA state- 
ment type line, the background colour bar extends 
across the entire colour line i.e., across all columns, as 
such a statement is enabled to extend across the entire 
line in one continuous string. 

As above stated the story editor table uses a two 
section screen since the story table is too wide to fit on 
a single screen. The screen shown in Fig. 17 is the one 
normally displayed. The right four columns shown in Fig. 
18 are used when working with display coordinates of 
partial image portions. As above stated, when the table 
of Fig. 1 7 is displayed, and the cursor is in the right most 
column (SIZE column) the depression of the keyboard 
tab key to cause the cursor to move to the initial character 
position of the next column in the story table (IMAGE. X, 
Y) simultaneously causes the screen to change to dis- 
play the table shown in Fig. 18. The action also occurs 
when a last character entry in the size column is followed 
by a right arrow key depression causing the next column 
filed to be ready for data input. 

Whenever the story table is displayed, the table is 
enabled for entry. Thus, wherever the cursor is posi- 
tioned at the moment, the depression of a character key 
on the keyboard causes that character to be inserted at 
that spot. The left-most column (Statement) will contain 
one or more commands from the AVA language. If the 
line is a screen-type line, the commands will generally 
be one of show, paste, clear or colour. If the line being 
inserted into the table is an audio type line, the command 
will generally be play. These commands can be seen in 
Fig. 42 along with the other AVA commands available for 
use in the system. If a command is inserted in the State- 
ment column, followed by the name of an image, it is that 
image which is commanded to be displayed. The clear 
command causes the screen to go to black and the paste 
command, followed by an image name causes one im- 
age to be superimposed over a previously shown image. 
The play command for a sound type line causes an audio 
object, whose name follows the command, to be played. 

Along the top of the columns in the story editor ta- 
bles, there are two lines, one labelled "screen" and the 
other "sound". Depending upon the particular command 
found in the Statement column, one or the other of the 
screen or sound lines is enabled. Thus, if a play state- 
ment exists in the Statement column, it is the sound line 
which is active, whereas rf a show statement is present 
in the Statement column, the screen line is active for the 
line on which the show command is emplaced. 

For lines where screen type statements are present, 
the Method column determines the dissolve technique 
used to bring the image referred to in the current line onto 
the screen. An entry to this field, and to the remaining 
fields in the story editor table can be keyed in, rotated in, 
or selected from the pull-down list shown in Fig. 1 9. As 
will be seen from a review of the various methods of dis- 
play, a number of techniques are available to cause a 
picture to move on or move off a screen. For instance, 



the checker method involves the current image being 
brought onto the screen in a series of random boxes in 
a checkerboard appearance. The explode method caus- 
es the current image to be unfurled onto the screen in 

s four directions simultaneously, etc. 

The Dir column determines the direction of a dis- 
solve method e.g. up, down, right, left, etc. (See Fig. 20). 
The next column to the right i.e. Line, provides a colour 
line along an incoming edge of the current image to sep- 

10 arate it from the prior image that is being overlayed or 
moved off the screen. This entry enables the colour of 
that line to be chosen. The Time column entry deter- 
mines the amount of time the dissolve method is to take 
from the start of the display to its completion. The time 

is values which can be chosen are shown in Fig. 22 and 
vary from immediate to 8 seconds. 

The Wait entry column determines the amount of 
time that processing is to wait, after completion of a cur- 
rent dissolve, until it advances to the next story line. Wait 

20 valuescanbeatimevalue (in seconds and tenths of sec- 
onds), an audio synchronisation label or time indication, 
the term "input" which requires waiting for input respons- 
es, a user actuated enter signal, etc. The possible entries 
here are shown in Fig. 23. The term key causes process- 
es ing to stop after completing the dissolve and to wait until 
a key (cursor or any other key) is depressed. An inserted 
time value causes the processing to stop for that specific 
length of time. Once the inserted time has expired, the 
processing proceeds onward to the next story line auto- 

30 matically When A or B is at the start of a wail entry, it 
indicates to wait for an audio synchronisation point, ei- 
ther from audio channel A or channel B. If the A or B is 
followed by a number, then the audio synchronisation 
point is a time value - that many seconds or tenths of 

35 seconds into the audio file. The time value is measured 
from the very beginning of the audio file, not from where 
the audio started playing. If the A or B at the start of the 
wait entry is followed by a character entry instead of a 
numeric value, then the entry is interpreted as audio 

40 synch label. At run time, these labels are equated to time 
values by the story editor. Thus, waiting for an audio 
synch label is identical to waiting for an explicit audio 
time, except the user has the convenience of using a la- 
bel reference instead of having to try to remember a par- 

45 ticular time value. For instance, an entry of "A Horn" 
specifies a wait until the synch point in the audio file 
named "horn" is reached. This column enables the visual 
portion of the presentation to be very precisely synchro- 
nised with the audio presentation. 

so The next column i.e. Size, determines whether the 
dissolve method image for the image object named on 
the current line is to be full screen or only for a part of 
the screen. As shown in Fig. 24, two choices are availa- 
ble i.e., full or part. If part is chosen, then entries are re- 

55 quired in one or more of the three right-most columns in 
Fig. 18. Image X, Y column determines where the upper 
left comer of the partial display area is to come from off 
the current image. For example, an X, Y entry of 20, 30 
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would position the upper left corner of the partial view at 
a position 20% across the screen from the left edge and 
30% down from the top edge. Entries in the WDTH, 
HGHT columns work exactly in the same fashion as is 
used for the Image X, Y coordinates. The difference is 
that these coordinates define the Width and Height re- 
spectively of the partial area For example, an entry such 
as 30, 40 defines a partial area that is 30% of the screen 
size wide and 40% of the screen size in height. These 
dimensions are relative to the upper left corner of the par- 
tial area defined in the Image X, Y column. The Screen 
X, Y column works in the same fashion as used for the 
image X, Y coordinates. The difference here however is 
that these coordinates define the upper left comer where 
the partial area is actually to be displayed on the screen. 
For example, an entry such as 80, 60 defines a partial 
target area that has its upper left comer, X coordinate at 
a position 80% across (from the left edge) and its upper 
left comer, Y coordinate 60% down from the top of the 
screen. 

As can now be seen, for a screen type tine, entries 
in the columns of the story editor table enable the method 
of presentation image to be totally defined, including its 
method of presentation and dissolve, its duration of pres- 
entation, etc. Similar entries, where a sound-type line is 
present, enable a sound object to be accordingly edited. 

For a sound type line, the Method column has three 
possibilities, fade in, fade and fade out {shown in Fig. 
28). The term fade", fades up the sound at its inception 
and fades down the sound at its completion. "Fade-in" 
just controls the commencement portion of the sound, 
and "fade-out" controls the termination portion of the 
sound. The channel column enables selection of one or 
the other of channels A or B (see Fig. 29). The Volume 
column enables an entry (Fig. 30) to be inserted (for a 
play statement) to set the volume level for the respective 
play activity. The Time column entry (for sound type 
lines) determines the amount of time the audio dissolve 
methods of fade, fade in or fade out take. As shown in 
Fig. 31 , a number of adjustments are available to these 
presentation methods. The Wait column entry deter- 
mines the amount of time that processing is to wait after 
completion of a current play statement, until it advances 
to the next story line. Wait values can be a time value, in 
seconds, etc., an audio synchronisation point, the term 
INPUT (which requires waiting for input responses and 
an entry actuation), the term TRIGGER which defines 
waiting for a cursor field to be activated etc.. The Wait 
column functions are similar for sound type lines as for 
screen type lines and operate much the same as de- 
scribed above for screen type lines. . 

The Size column entry determines whether the play- 
ing of an audio file is to be from the very start to the very 
end of the file (full), or whether the playing is to start and 
end at other points in the file (part), (see Fig. 33) 

The Sound Beg column entry applies only to a play 
statement such as PLAY followed by the name of an au- 
dio object. It allows the specific start point to be desig- 



nated - one that can be other than the start of the audio 
file. The entry may be an audio file synch label or a time 
in seconds and tenths of a second. The Sound End entry 
works in exactly in the same fashion as for Sound Beg 

5 except it designates where to stop playing the audio file. 
These are shown in Figs. 34 and 35. 

Turning now to Figs. 36-39 the various action bar 
functions associated with the story editor table will be 
briefly discussed. Many of the functions are identical to 

io those used with the audio editor table. The Prompt action 
has been considered above and will not be further de- 
scribed here. The Tell action (Fig. 36), is a command 
which begins the telling of a story, starting at the current 
cursor location in the story table. Tell generally continues 

is until the last line of the story is reached, a stop statement 
or break point is reached or an escape key is pressed. 
As soon as Tell reaches a first screen type line, it search- 
es upwardly in the story table until it encounters a show 
or paste statement that has an image name associated 

20 with it. This provides the starting image display so that 
display of the screen-type line on which Tell resides, can 
be done with whatever image-to-image transition meth- 
od is currently in effect for this line. Next, Tell determines 
what image to display for the current line. If the current 

25 line, screen-type statement has an explicit file name or 
variable name, then that is used. If it does not, then the 
same image used for the starting display becomes the 
display for this line. Tell then continues to sequence 
down the story lines executing the commands as found. 

30 The remaining Tett actions of Fig. 36 are self explanatory 
and will not be further described here. 

The Edit actions (see fig. 37) enable an entered story 
line to be altered, copied, etc. as determined by the user. 
The view action commands enable various portions of 

35 either a source image or target image to be selected for 
alteration or provide the ability to modify the image while 
listening to the audio or edit the image with the audio. 

The Print, File, Exit and Help actions shown in Figs. 
38 and 39 are further actions available to the user to per- 

40 form the indicated functions. 

Figs. 40, 41 and 42 as aforestated, show the various 
table prompts usable in the creation of a story table. As 
shown in Fig. 41 and 42, a multiplicity of various functions 
and commands are available for insertion in the story ta- 

45 ble to enable logical and arithmetical processing to occur 
during the play of a story. Furthermore, these functions 
and commands enable interactivity to be inserted into the 
story, so that it awaits a user's reaction to a portion of the 
story before it will proceed, branch or otherwise alter its 

so presentation. 

STORY AUTHORING 

Turning now to Fig. 43, a story table is illustrated 
ss which, in combination with the audio table shown in Fig. 
1 4 enables the presentation of a synchronised audio vis- 
ual presentation. Note that comments are inserted on 
lines 1-3 and 16-18, with the "/*" enabling the comments 
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to overwrite the column division lines. 

The story begins on line 11 of the story file with a 
showing of an image labelled 'Intro'. The column inser- 
tions on line 11 indicate that the image fades in with the 
fade occurring from right to left and the fade taking 0.5 
seconds to occur. The full image entitled "Intro" is shown. 

At line 12, a command to load the image "sailors" is 
executed and then, on line 1 3, the command to "wait" for 
the depression of a key is found. Upon the user depress- 
ing a key, the cursor moves to line 19, and immediately 
causes play to commence of the Retail 1 audio file from 
channel A, at 100% volume. The "A 2" entry in the Then 
Wait column indicates that an audio synchronisation 
point of 2 seconds into the Retail 1 audio file is to be used 
as the time to move the cursor in Fig. 43 to line 20 and 
to perform the next command. 

At the synchronisation point A 2, the cursor moves 
to line 20 and the system executes of the command 
•show ALLBLACK" which image is brought to the screen 
using a fade which occurs over a two second interval 
(see time column). The ALLBACK image is held in place 
for 0.5 seconds and the cursor then proceeds to scroll to 
line 21 . There the show command "navyhat" is executed 
using a checker type of presentation to present the im- 
age, with one second then being waited until the cursor 
is scrolled to line 22. The "Show Sailors" command caus- 
es the "Sailors" image to be displayed using an "explode" 
technique, with the image coming into the centre from a 
vertical direction. A red broader line appended to the im- 
age is commanded by the entry in the Line column, and 
the entry "6" in the time column indicates that it will re- 
quire six seconds for the full image to be developed. The 
Sailors image stays in place until the 21 st second of time 
is sensed from the A audio channel ("A 21 " entry in the 
"then wait "column). 

Story editor lines 23-26 continue as shown. At line 
27, the screen goes to black using a stripe presentation 
which occurs over 0.8 seconds. Then, the entry in the 
"Then, wait" column of A NORD indicates that only upon 
the occurrence of the NORD synchronisation label from 
the retail 1 audio file shown in Fig. 14, will the presenta- 
tion be allowed to continue. Upon that synchronisation 
label being sensed, a logo is then shown (see line 29) 
using a partial view as defined in the four right most col- 
umns of the story file. The presentation ends as defined 
in lines 30-32. 

It can thus be seen that through the use of the audio 
editor table and the story editor table, a user is enabled 
to construct a synchronised, audio/visual presentation. 
Then, during run time, the PC operates to sequentially 
access the statements in conjunction with the playing of 
the audio file to present the story. 



Claims 

1. A method of creating and performing a synchro- 
nised, audio/visual presentation in a data process- 



ing system having audio and visual input/output, the 
method comprising storing a plurality of visual 
images; 

s and characterised by: 

displaying a table representing an audio 
sound-track together with corresponding time 
indications, and entering synchronisation labels 
at selected time indications; 

10 entering a command for displaying selected 

stored visual images and associating said com- 
mand with a synchronisation label in the audio 
sound-track table; and 

executing the audio/visual presentation by play- 
is ing the audio sound-track, and responsive to 

recognising a synchronisation label therein, 
invoking the command associated with said 
synchronisation label to display the selected 
stored visual image. 

20 

2. A method as claimed in claim 1 , further including 
entering into the table, volume control commands to 
enable volume changes in the audio sound-track. 

25 3. A method as claimed in claim 1 or 2, wherein the 
table is provided with a list of allowable actions per- 
formable on the audio/visual presentation, only a 
listed allowed action being enabled in response to a 
user selection thereof. 

30 

4. A method as claimed in any preceding claim, further 
comprising the steps of: 

playing the audio sound-track; 
35 and displaying continuing sound indications of 

the audio sound track along one dimension of 
the editor screen, accompanied by discrete time 
of play indications. 

40 5. A method as claimed in claim 4, further comprising 
inserting narrative text adjacent the sound indica- 
tions and time of play indications on the screen when 
the narration is to occur. 

45 6. A method as claimed in any preceding claim, further 
comprising inserting a command in the table to delay 
further processing until a user response is received. 

7. A method as claimed in claim 6, further comprising 
so inserting a command to perform a non -screen, 

non-audio process and to await its completion until 
proceeding with further commands. 

8. A method as claimed in any preceding claim, further 
ss including displaying a tabular representation of the 

audio/visual presentation, including a column for 
command statements, columns for visual presenta- 
tion function indications, and columns for synchro- 
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nisation labels and time indications; enabling a user 
to insert in the tabular representation the command 
statements, synchronisation labels and time indica- 
tions and to select and insert the visual presentation 
functions; whereby during execution of the 
audio/visual presentation, the visual images are pre- 
sented in accordance with the function indications, 
time indications and synchronisation labels inserted 
in the tabular presentation. 



Patentanapruche 

1. Ein Verfahren, urn eine synchronisierte audiovisu- 
elle Darstellung in einem Datenverarbeitungsgerat 
zu erstellen und auszufuhren, das audiovisuelle 
Ein-/Ausgange hat, wobei das Verfahren die Spei- 
cherung einer Vielzahl von visuellen Bildern enthalt; 

und dadurch gekennzeichnet wird, daB 
eine Tabelle angezeigt wird, die einen 
Audio-Soundtrack zusammen mit entsprechen- 
den Zeitangaben darstellt und Synchronisie- 
rungskennungen in ausgewahlten Zeitangaben 
eingibt; 

ein Befehl zur Anzeige von ausgewahlten, 
gespeicherten visuellen Bildem eingegeben 
wird, wobei dieser Befehl mit einer Synchroni- 
sierungskennung in der Audio-Soundtrackta- 
belle verbunden wird; und' 
die audiovisuelle Darstellung ausgefuhrt wird, 
indem der Audio-Soundtrack wiedergegeben 
wird und auf die Erkennung einer in der Tabelle 
befindlichen Synchronisierungskennung rea- 
giert, wobei der Befehl aufgerufen wird, der mit 
der Synchronisierungskennung verbunden ist, 
urn das ausgewahlte, gespeicherte, visuelle 
Bild anzuzeigen. 

2. Ein Verfahren wie in Anspruch 1 angemeldet, das 
auBerdem die Etngabe von Volumensteuerungsbe- 
fehlen in die Tabelle enthalt, urn Volumenanderun- 
gen im Audio-Soundtrack zu ermoglichen. 

3. Ein Verfahren wie in Anspruch 1 oder 2 angemeldet, 
wobei die Tabelle mit einer Liste von zulassigen 
Aktionen geliefert wird, die in der audiovisuellen 
Darstellung durchfuhrbar sind und nur eine aufgeli- 
stete, zugelassene Aktion in der Lage ist, auf eine 
vom Anwender getroffene Auswahl zu reagieren. 

4. Ein Verfahren wie in irgendeinem vorhergehenden 
Anspruch angemeldet, das auBerdem Schritte ent- 
halt, um 

den Audio-Soundtrack wiederzugeben; 

und die kontinuierlichen Sound-Angaben des 



Audio-Soundtracks entlang einer Dimension 
des Editor-Bildschirms zusammen mit einzel- 
nen Angaben zur Wiedergabezeit anzuzeigen. 

s 5. Ein Verfahren wie in Anspruch 4 angemeldet, das 
auBerdem erzahlenden Text in der Nahe der 
Sound-Angaben und der Angaben zur Wiedergabe- 
zeit im Bildschirm einf ugt, wenn die Erzahlung aktu- 
ell vorkommt. 

10 

6. Ein Verfahren wie in irgendeinem vorhergehenden 
Anspruch angemeldet, das auBerdem einen Befehl 
in die Tabelle einf ugt, um die weitere Verarbeitung 
zu verzogem, bis der Anwender antwortet. 

75 

7. Ein Verfahren wie in Anspruch 6 angemeldet, das 
auBerdem einen Befehl einfugt, um eine 
Non-Screen-, Non-Audio-Verarbeitung durchzufuh- 
ren und bis zu dessen AbschluB zu warten, bevor 

20 weitere Befehle verarbeitet werden. 

8. Ein Verfahren wie in irgendeinem vorhergehenden 
Anspruch angemeldet, das auBerdem enthalt: die 
Anzeige einer tabellarischen Darstellung der audio- 
es visuellen Presentation, eine Spalte fur Befehlsan- 

weisungen, Spaften fur visuelle Darstellungsfunkti- 
onsangaben und Spatten fur Synchroniserungsken- 
nungen und Zeitangaben; wobei es dem Anwender 
ermoglicht wird, die Befehlsanweisungen, Synch ro- 

30 nisierungskennungen und Zeitangaben einzufugen 
und die visuellen Darstellungsfunktionen auszu- 
wahlen und einzufugen; wobei wahrend der Ausfuh- 
rung der audiovisuellen Darstellung die visuellen Bil- 
der gemaB den Funktionsangaben, den Zeitanga- 

35 ben und den Synchronisierungskennungen, die in 
die tabellarische Darstellung eingefugt wurden, pra- 
sentiert werden. 



*o Revendlcatlons 

1. Procede pour creer et executer une presentation 
audio/visuelle dans un systeme de traitement de 
rinformation comprenant des entrees/sorties audio 
45 et vid6o, ledit proced6 consistant a memoriser une 
pluralite d'images visuelles et etant caracterise par 
les etapes consistant a : 

afficher une table representant une bande son 
so audio avec des indications temporelles corres- 

pondantes, et entrer des etiquettes de synchro- 
nisation a des indications temporelles selection- 
nees; 

55 saisir une commando pour afficher des images 

visuelles memorisees selectionnees et associer 
ladite commande a une etiquette de synchroni- 
sation dans la table de bande son audio: et 
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2. 



3. 



exScuter la presentation audio/vis ue lie en lisant 
la bande son audio et, en reponse a la recon- 
naissance d'une etiquette de synchronisation 
dans celle-ci, en appelant la commande asso- 
ci6e avec ladite etiquette de synchronisation $ 
pour afficher I'image visuelle memoris6e s6lec- 
tionn6e. 

Proced6 selon la revendication 1 comprenant, de 
plus, I'etape consistant a introduire dans la table des io 
commandes de contrdle de volume pour valider des 
changements de volume dans la bande son audio. 

Proced6 selon la revendication 1 ou 2, dans lequel 
la table comprend une liste d'actions autorisables is 
pour execution sur la presentation audio/visuelle, 
seule une action autoris£e Iist6e pouvant etre vali- 
dee en reponse a la selection de celle-ci par I'uttli- 
sateur. 



4. 



Procedd selon i'une quelconque des revendications 
pr6cedentes comprenant, de plus, les etapes con- 
sistant a : 



20 



mande, des etiquettes de synchronisation et des 
indications temporelles, et de selectionner et d'ins6- 
rer des fonctions de presentation visuelle; caracte- 
rise en ce que, au cours de I'execution de la presen- 
tation audio/visuelle, les images visuelles sont pre- 
sentees conform6ment aux indications de fonctions, 
aux indications temporelles et aux etiquettes de syn- 
chronisation inserees dans la presentation tabu- 
late. 



lire la bande son audio; et 
afficher des indications audio en continu en pro- 
venance de la bande son audio le long d'une 
des dimensions de P6cran d'6dition, accompa- 
gnees par des indications de temps de lecture 
discretes. 



2S 



6. 



Precede selon la revendication 4 comprenant, de 
plus, Pinsertion d'un texte narratif a c6t6 des indica- 
tions audio et des indications de temps de lecture 
sur I'ecran lorsque la narration doit avoir lieu. 35 

Precede selon I'une quelconque des revendications 
prec6dentes comprenant, de plus, ('insertion d'une 
commande dans la table pour retarder le traitement 
suivant jusqu'a ce qu'une reponse de I'utilisateur ait *o 
ete recue. 



Precede selon la revendication 6 comprenant, de 
plus, insertion d'une commande pour executer un 
traitement non-6cran, non-audio, et pourattendre la 
fin de son execution avant de passer aux commande 
suivantes. 



45 



Precede selon I'une quelconque des revendications 
precedentes comprenant, de plus, les etapes con- 
sistant a : afficher une representation tabulaire de la 
presentation audio/visuelle, comprenant une 
colonne pour des instructions de commande, des 
colonnes pour des indications de fonctions de pre- 
sentation visuelle, et des colonnes pour des etiquet- 
tes de synchronisation et des indications temporel- 
les; permettre a un utilisateur d'insdrer dans la 
representation tabulaire des instructions de corn- 



so 



55 
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